Semantic-Guided Selective Representation for Image Captioning

Grid-based features have been proven to be as effective region-based in multi-modal tasks such visual question answering. However, its application image captioning encounters two main issues, namely, noisy and fragmented semantics. In this paper, we propose a novel feature selection scheme, with Relation-Aware Selection (RAS) Fine-grained Semantic Guidance (FSG) learning strategy. Based on the grid-wise interactions, RAS can enhance salient regions channels, suppress less important ones. addition, process is guided by FSG, which uses fine-grained semantic knowledge supervise process. Experimental results MS COCO show proposed RAS-FSG scheme achieves state-of-the-art performance both off-line on-line testing, i.e., 134.3 CIDEr for testing 135.4 of MSCOCO. Extensive ablation studies visualizations also validate effectiveness our scheme.